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For better or worse, virtual imaging displays are with us in the form of narrow-angle 
combining-glass presentations, head-up displays (HUD), and head-mounted projections of wide- 
angle sensor-generated or computer- animated imagery (HMD). All of our military and civil avia- 
tion services and a large number of aerospace companies are involved in one way or another in a 
frantic competition to develop the best virtual imaging display system. The success or failure o 
major weapon systems hangs in the balance, and billions of dollars in potential business are at 
stake Because of the degree to which our national defense is committed to the perfection of virtual 
imaging displays, a brief consideration of their status, an investigation and analysis of their prob- 
lems, and a search for realistic alternatives are long overdue. 


CURRENT STATUS 


All of our currently operational tactical fighter aircraft are equipped with HUDs. Helicopters 
are navigated and controlled, and their weapons are delivered, with a variety of imaging displays 
including, in addition to HUDs, both panel-mounted and head-mounted image intensifies and 
forward-looking infrared (FLIR) and low-light TV displays. Even some strategic aircraft and a 
few commercial airliners contain virtual imaging displays. A new generation of remotely pi ote 
vehicles (RPV) are intended to be flown by reference to wide-angle but relatively low-resolution 
sensor imagery presented stereoscopically by head-mounted binocular displays. And Detroit is 
about to offer HUDs for cars. 


THE TROUBLE WITH HUDS AND HMDS 


As for the operational problems, about 30% of tactical pilots report that using a HUD tends to 
cause disorientation, especially when flying in and out of clouds (Barnette, 1976; Newman 1980). 
Pilots frequently experience confusion in trying to maintain aircraft attitude by reference to the 
HUD’s artificial horizon and "pitch-ladder” symbology, particularly at night and over water, and 
there are documented cases of airplanes becoming inverted without the pilots' awareness (Kehoe, 
1985) Pilots have also reported a tendency to focus on the HUD combining glass instead of the 
outside real-world scene (Jarvi, 1981; Norton, 1981). The resulting myopia is a special case of the 
more general anomaly known as "instrument myopia" (Hennessy, 1975). 


Misaccommodation of the Eyes 

Whatever the cause, it is a repeatedly observed experimental fact that our eyes do not automati- 
cally focus at optical infinity when viewing collimated virtual images, but lapse inward toward t eir 
dark focus, or resting accommodation distance, at about arm’s length on average (Hull, Uiil, and 
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Roscoe' andPetitM^Rm PP*’ P P COe ' 1988; P™" and Ehrlich - 1986 ^ Ra "“'e. 

?^’ , etltt ’ 1 ^ 80 - ^ perceptual consequence of positive misaccommodation is that the 

whole visual scene shrinks in apparent angular size. This shrunken appearance causes distant 

objects to be judged farther away than they are, and anything below the line of sight, such as the 

(Lsc^ f 1984^98 n 5) r “ nmWay ’ appeafS hlgher ±an U reall y is relative to the horizon 

The effect of the HUD optics is illustrated in figure 1. The experiment was conducted by Jovce 
and Helene Iavecchia at the Naval Air Development Center in Pennyslvania. A HUD was set ud 
on one rooftop and a "scoreboard” assembly with selectively lighted numerals of various sizes was 
mounted on top of another building 182 m away and of about the same height. Observers were 

half thl° triak S r° reb0ard n , Um 5 erS aS thCy appeared and also numbers presented by the HUD on 
hdf the trials. Concurrently, the eye accommodation of the observers was measured with a polar- 
ized vernier optometer. p 

terr^IfhT 1 the 6 f °c al responses to the scoreboard numerals and the background 
errain beyond the scoreboard, with the HUD turned Off and with it turned On. In ei the/case the 
observers focal responses were highly dependent on their individual dark focus distances- in fact 
knowing each individual's dark focus accounted for 88% of the variance in focal reZnses unS 

d ‘ h H “PT Exduding 0b5crver 9 ’ whoK da * «*» 

shon of 1 m y ° nf mfinity ’ the avera S e for lire remaining nine emmelropes was 1.06 D, or just 

But the striking result shown in figure 1 is the fact that when the HUD was turned On for all 
nr s n erS ’ f ° CUS shlf ' ted inward an average of 0.02 D, or 50 m, to an average of 0.20 D 
to 0 47 D ex ^ luding 0bserver 9, the average inward shift was from 0.27 D, about 4 m ’ 

to 0.47 D, about 2 m. Although such shifts have little effect on the apparent clarity of the visual ’ 

Ltures y tremend ° US ^ ° n 1116 apparcnt Size ’ distance > a " d 


Accommodation and Apparent Size 

Despite wide individual differences among observers, the average apparent size of objects is 

HTnf eC ?D COrrelatd (l > °' 9) with the ^stance at which the eyes are focused (Benel 1979- 
^ ;, Gl ^ ii-and Roscoe, 1982; Iavecchia, Iavecchia, and Roscoe, 1983; Roscoe Olz^ and ^ ’ 

HUD svmboln lm ° nem ’ l 9 J 9 ^ Thus > the P osi dve misaccommodation induced by collimated 
HUD symbology can partially account for the fact that pilots flying airplanes or flight simulSors hv 

f^mn^l^M 1 ^ 13 h ma ^ in ^)Y SternS make faSt ap P roaches ’ 

(Campbell, McEachem, and Marg, 1955; Palmer and Cronn, 1973). g 

(19) while flying by reference to collimated HUDs (Morphew 19851 When flvino ° 
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Fixed-wing airplane pilots flying with HUDs also judge a target to be farther away and the dive 
angle shallower than they are, resulting in almost-always-fatal "controlled-flight-mto-t e-eiTain 
accidents. In the U.S. Air Force, such accidents have continued to occur at the rate of about one 
per month since FIUDs came into general use at the beginning of this decade. Two months ago 
(June 1987) an F-16 left a smoking hole in the ground, and last month it was an F-l 1 1 . 1 ne 
Navy's experience has been essentially the same. 


Optical Minification 


Misorientation and disorientation with panel-mounted and some head-mounted imaging dis- 
Dlavs are exacerbated by the fact that limited display size and the need to display the widest practi- 
cLoutside visual angles typically result in drastic optical minification, which adds to the perceptual 
minification caused by the misaccommodation. If the display area were not so limited an cou 
varied to accommodate the wide individual differences in dark focus 

side world could be magnified by appropriate amounts to neutralize each mchvidual s percep 
bias The average magnification required would be XI. 25 (Roscoe, 1984; Roscoe, Hasler, a 
Dougherty, 1966), but this value would be correct for only a portion of the population, possibly 

requiring stricter pilot selection. 


Image Quality 

Display minification and perceptual biases are two sources of error in human judgments of 
size, distance, and angular location, but there are other sources of error as well, namely, the van- 
able errors associated with adverse ambient viewing conditions (atmosphenc attenuation an 
reduced illumination), the limited resolution of cameras and display devices, and the further loss of 
resolution with image intensification. All of these factors serve to reduce contrast and detail the 
principal components of image quality, and the accuracy with which people can extract positions, 
rates, and accelerations relative to outside objects in the visual environment. 


DISPLAY ALTERNATIVES 


Because of the adverse effects of virtual images on eye accommodation, as well as the optical 
minification and poor image quality typically associated with sensor-generated displays, our 
judgments of spatial relations are simply not good enough to support complex flight missions as 
safely or effectively as we need. To date the advocates of virtual image displays have adaman y 
refused to acknowledge the implication of misaccommodation in the misonentanon and ^soi^nta- 
tion of pilots flying with HUDs. Instead they have attributed the problems primarily to the limited 
fields of view afforded by the combining glasses used with current systems. 

To address the limited-field-of-view problem, each of our ^tary seivices, including the 
Marines, is spending millions of dollars a year-to say nothing of the IR&D funds invested by 
private companies— to develop wide-angle, head-mounted imaging displays, in many cases cou- 
pling camera line-of-sight to head or eye orientation. Still clinging to the assumption that the eyes 
will focus collimated images at optical infinity, the advocates of head-mounted displays and 
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head-coupled sensors now promise that a pilot will be able to maintain geographic orientation and 
^ints'M(ftargets^ SmentS ° f diStanCCS ’ rates ° f doSure ’ and an S ular Sections to visible navigation 

To dispel any doubt that such promises will come true, designers of some sensor and display 
systems are delivering imagery from two cameras independently to the two eyes to provide stereo- 
scopic viewing (or even hyperstereo by exaggerating the interocular distance between the cameras) 
Many are convinced that stereo viewing will create an illusion of "remote presence" and thereby ' 
improve judgments of size, distance, and angular location sufficiendy to make it unnecessary to 
provide automatic sensors of vehicle positions and rates for navigation and obstacle avoidance. 

xpenence with head-mounted displays, whether binocular or biocular (both eyes receiving the 
same images), does not warrant these wishful thoughts. 6 

fro , m a variet y of experimental and operational contexts indicates that binocular judg- 
ments of size and distance are not markedly better than monocular judgments, except at very short 

judgment to threading a need1 ^- | n fact - Holway and Boring (1941) found monocular rize 
r b , re nearl > 'veridical than binocular judgments when good distances cues are pre- 
sent. In any case, the large bias errors in size, distance, and angular position judgments caused by 
saccommodation to virtual images would more than cancel any minor benefits of disparate 
images to the two eyes. p<u<llc 

In th . e absence of some strik ing breakthrough in human genetic engineering, the long-range 

head -« d SP la y s is "»> sood. Not only do our eyes refust to behave £ dis- 
p ay designers would like to believe, but the illusion of vection induced by the "streaming" of 

£l7r "Th penphei ^ °U wide - an S le views often leads to motion sickness, particularly with 
SvT P e 1 S ® nsors , and conse quent smearing of the images with head movements. Unfortu- 
HMDs ilTfhp f depe . ndence °? v jriual imaging displays for tactical missions (HUDs now and 
„ ., D . the fu . tu f e) has resulted in almost total suppression of research and development of more 

withtSS marnSon SPlayS ° f S “ mCien ' a " gUlar ““ ‘° provide the needed fidds ° f 


WHAT CAN BE DONE 


3fH n dmiS ? e gen f lc engmeenng approach, there are still several reasonable courses of 
action. In the short run, these include (1) trying to "fix" the HUD optics to compensate for the 
misaccommodation that leads to misorientation, and (2) modifying the ambiguous HUD symbol 
w .ha, leads to attitude reversals and subsequent disorientation. In the lon^r SS 

l Tv „ n SC ,' 1PPr0a f a ," d concentra,e on lar ge. integrated forward-looking and downward- 
looking direct-view dtsplays in which computer-animated flight atdtude, guidance, and prediction 
symbology ,s superposed on sensor-generated real-world imagery. predicnon 


Fixing the HUD 


HQ8*\ d r ? 10tS ? f0C , US at optlcal lnfim ty when viewing virtual images, Norman and Ehrlich 
! 86) in lsra f inn ' odl ’ ce d a negative focal demand of -0.5 D with the desired result, although 
there were wide individual differences in responses as a function of individual dark-focus § 
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distances. Thus, the first experimental fix should be the addmonof variable optical refraction to 
offset each individual pilot's inward focal lapse induced by the HUD s virtual images^ Turning 
HUD On would require a key coded to select the pilot's specific correction based on the dark 
focus. At this time, no one can be sure how successful this fix will be, but it must be tried. 

Almost as important is the complete redesign of HUD symbology. Just how complicated and 
confusing it is can be appreciated from the estimate of an Army Instructor Pilot that an average 
student helicopter pilot requires 200 hr of simulator and flight training to master the gaggle of 
symbols (personal communication). Furthermore, the attitude presentation m fixed-wing airp a 
is conducive to horizon and pitch-ladder control reversals that result in disorientation and 
"graveyard spirals" at night and in marginal weather. At the very least, a frequency- separated p 
dieted flightpath "airplane" symbol that banks and translates in immediate response and in the sa 
direction as control inputs should replace the present velocity vector and acceleration symbology 
(Roscoe, 1980, Ch. 7; Roscoe and Jensen, 1981). 


Presenting the Big Picture 

If head-mounted, wide-angle imaging displays are ever to be safe 
minification of the outside world will have to be compensated for by individually «lec«£te ophod 
magnification, or the eyes will have to be induced to focus at or near optica infinity as in the case 
of HUDs. Neither approach will be simple. Furthermore, the whole virtual image display concept 
depends on a gross reduction, rather than any increase, in the weight of any head-mounted device 
to be used in a high-Q environment. All things considered, it is surely premature to give up on 
direct-view, panel-mounted displays. 

Large, integrated, direct-view displays offer many advantages in terms of visual performance 
as well as ease of achievement and lower cost. Eyes focus real images m °re accurately than virmai 
images (Hull, Gill, and Roscoe, 1982; Iavecchia, Iavecchia, and Roscoe, 1988, Randle, Roscoe, 
and Petitt 1980). Although many with 20/20 vision cannot focus out to optical infinity, a 

focus at the distance of cockpit instrument panels. Thus, although magnification 
of sensor-generated or computer-animated images of the outside world will be required, as it is 
with direct-view projection periscopes (Roscoe, 1984; Roscoe, Hasler, and Dougherty, 1966), a 
single, fixed-magnification factor of about XI .25 will suffice for most emmetropes. 

To make room for large forward-looking and downward-looking (and possibly sideways- 
looking) displays, a lot of single- variable dedicated instruments and controls will have to be 
replaced by insets that appear selectively on the large displays as a function of the mission phase 
aircraft configuration, mode of operation, weather and traffic, system malfunctions and in the case 
of military aircraft, weapon selection. Furthermore, with the ever-increasing complexity of aircraft 
systemsand military missions, many future airplanes— despite their high degrees of automation - 
will require at least two pilots with a redistribution of functions and available information. 

In the military there will always be a heavy premium on being able to take advantage of what- 
ever is visible to the naked eye. However, trying to combine synthetic imagery with contact visi- 
bility compromises both, and a strong case can be made for distributing operational functions and 
information sources between an "inside" pilot and an "outside pilot- The insi e pi o wou 
mally do all the flying in instrument meteorological conditions (1MC) and most of the flying unde 
visual meteorological conditions (VMC), using a direct-view, wide-angle projection periscope and 
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toe large, panel-mounted pictorial displays sunrounding toe pilot deep inside toe airplane The 
outside pilot would use his or her eyes to supplement toe imaging sensors, do most of toe com- 
mumcatmg and procedural housekeeping, and fly any maneuver that requires direct contact 
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AVERAGE RESPONSES TO SCOREBOARD 
AND TERRAIN CONDITIONS: 
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Figure 1- Average focal responses to the scoreboard and the terrain conditions with HUD On and 
Off, plotted against each individual's dark focus. 




